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US-PAT-NO: 6134343 



DOCUMENT-IDENTIFIER: US 6134343 A 

TITLE: System or method for detecting defect within a 

semi-opaque enclosure 

KWIC 

Abstract Text - ABTX (1): 
A method and system detect for the presence of a defect or a non-conforming 
object of unknown shape, configuration, and location within a semi-opaque 
enclosure. The semi-opaque enclosure has a pattern which is visible on at 
least a first of its external surfaces. The semi-opaque enclosure may 
comprise, for example, a labeled plastic or glass bottle, and the visible 
pattern may be print and graphical information provided on the bottle's label. 
A first digital image is captured from first channel light reflected off the 
first external surface of the enclosure; and a second digital image is captured 
from second channel light navigating the object inside the enclosure and 
emanating from the first external surface of the enclosure. A difference image 
is formed substantially devoid of information representing the visible pattern 
on the first external surface, by subtracting one of the first and second 
digital images from the other. Additional segmentation processing may be 
performed on the difference image in order to better identify the presence of a 
defect or non-conforming object within the semi-opaque enclosure. 

Application Filing Date - AD (1): 
19990407 

TITLE -TI(1): 

System or method for detecting defect within a semi-opaque enclosure 

Parent Case Text - PCTX (5): 
U.S. patent application Ser. No. 08/621,189, entitled "Machine Vision 
Methods for Inspection of Leads on Semiconductor Die Packages," filed in the 
name of Sanjay Nichani on Mar. 2 1 , 1996; 

Parent Case Text - PCTX (6): 

U.S. patent application Ser. No. 08/621,190 entitled "Machine Vision 
Methods for Inspection of Semiconductor Die Surfaces," filed in the name of 
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Sanjay Nichani on Mar. 21, 1996; 

Brief Summary Text - BSTX (6): 

Computer vision, otherwise referred to as image processing, involves the 
extraction of vision-related information by obtaining signals representing a 
scene and performing image signal processing on those signals. Applications of 
computer vision (image processing) techniques include character recognition, 
industrial inspection of manufactured items, robot guidance systems, radiology, 
remote sensing, and so on. 

Brief Summary Text - BSTX (13): 

Golden Template Comparison (GTC) is an image processing method that has been 
used to detect flaws and defects in two-dimensional scenes of an inspected item 
which are highly repeatable and do not suffer from geometric distortion. Such 
scenes are common in semiconductor production and graphic arts applications. 
Generally, GTC involves the comparison of a test image to an image of a known 
good scene referred to as a template image. More specifically, the test image 
is subtracted from the template image, and differences between the two images 
are observed in order to determine whether a flaw or defect is present within 
the inspected item. 

Brief Summary Text - BSTX (14): 

Generally, GTC comprises two main phases: training and inspection . During 
training, the template image is constructed by sampling a plurality of images 
each of which represents a scene of an item absent defects . During inspection, 
the test image, which represents the scene of the item to be tested (the 
inspected item), is then compared to the template image. More specifically, 
the images are subtracted to form a difference image. Thresholding is 
performed on the difference image to produce a resulting binary image called an 
error image containing either background (healthy) pixels or defect pixels 
which represent flaws. The error image is then analyzed in order to determine 
whether the tested item should be labeled as an "accept" or "fail" item. The 
analysis that may be performed on the error image may include counting defect 
pixels found in the error image and performing a blob analysis on the defect 
pixels. The results of the analysis may be stored in a data structure called a 
results structure. 

Brief Summary Text - BSTX (15): 

A. Issues Concerning the Use of Golden Template Comparison to Inspect the 
Inner Side- Wall of a Labeled Bottle 

Brief Summary Text - BSTX (17): 
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In order to produce (with thresholding) an error image which effectively and 
accurately represents defect pixels separate from healthy pixels, pixels within 
the difference image must have a sufficiently high gray level value. In order 
for this to occur, defects must give rise to a change in image intensity in the 
test image. In graphic arts applications, defects are typically defined in 
terms of changes in scene reflectivity, which result in a change in image 
intensity in the test image. Semiconductors are inspected to determine if they 
contain surface contaminants that can result in a subsequent device failure. 
Such surface contaminants frequently cause changes in scene reflectivity which 
thereby change the image intensity of the test image. 

Brief Summary Text - BSTX (18): 

In each of these situations, the defect is externally- visible. A problem 
arises, however, when inspecting semi-opaque enclosures (e.g., when inspecting 
the inner side-wall of a labeled bottle) to determine whether they contain 
unwanted objects ( defects ). In that application, the defect is not externally 
visible. Therefore, when inspecting the inside of semi-opaque enclosures for 
defects (non-conforming objects), it is difficult to obtain a test image that 
exhibits a change in image intensity due to the presence of a defect . 

Brief Summary Text - BSTX (19): 

(2) Gray-level Defect Criteria 

Brief Summary Text - BSTX (20): 
In order to produce an error image which accurately distinguishes defect 
pixels from background (healthy) pixels, thresholding may be performed in 
accordance with appropriate gray-level defect criteria. Various thresholding 
methods have been proposed which perform thresholding in accordance with 
different criteria. A simple type of defect criteria is photometric defect 
criteria which classifies pixels as background (healthy) pixels or defect 
pixels based upon image intensity alone. Other defect criteria (e.g., based on 
the shape formed by a group of adjoining defect pixels) can be utilized to 
guide the thresholding process. 

Brief Summary Text - BSTX (21): 

(3) Geometric and Morphological Defect Criteria 

Brief Summary Text - BSTX (22): 

The binary error image may be further processed to correct (replace) 
improperly classified pixels. For example, the binary error image may comprise 
defect pixels falsely classifying healthy pixels as defects . Accordingly, the 
binary image may be processed by imposing additional criteria, e.g., based, 
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upon the size and/or shape of a set of adjoining defect pixels. Simply put, 
defect pixels formed in the error image from small intensity variations in the 
test image may be ignored (i.e., replaced with healthy pixels) unless they 
represent a defect which is of a certain size and/or shape (e.g., if the defect 
is at least several pixels in length or diameter). Geometric and morphological 
defect criteria are disclosed by William M. Silver and Jean Pierre Schott in a 
document entitled "Practical Golden Template Comparison," provided by Cognex 
Corporation, the content of which is hereby incorporated herein by reference in 
its entirety. 

Brief Summary Text - BSTX (23): 

The usefulness of defect criteria (such as geometric and morphological 
defect criteria) can be significantly diminished when a distorted test image is 
acquired, e.g., by a conventional bottle inspection arrangement, as shown in 
FIGS.7A-7C. 

Brief Summary Text - BSTX (24): 

A conventional labeled-bottle visual inspection system is shown in FIG. 7A 
which comprises a vertically-arranged, elevated camera 200 having a wide-angle 
lens 200a pointed directly into an upper opening of a bottle 202. Bottle 202 
has a semi-opaque label 203 surrounding its outer side- wall surface 210, and is 
positioned over back lighting 206. A defect 204 is depicted in FIG. 7A with 
dotted lines, because defect 204 is not visible from outside of bottle 202 due 
to the opaque nature of label 203. Defect 204 is on an inner side- wall surface 
208 of bottle 202, behind label 203. FIG. 7B comprises a cut-away view of 
bottle 202, in which inner side-wall surface 208 is exposed, and defect 204 is 
viewable. The view provided in FIG. 7B directly faces defect 204. As 
depicted, defect 204 is dark in appearance and rectangular in shape. 

Brief Summary Text - BSTX (25): 

FIG. 7C shows a 2D test image 201 which is acquired by camera 200 with the 
use of a wide-angle lens 200a. In order for the complete inner side-wall 
surface 208 which spans the entire area of label 203 to be inspected, 
wide-angle lens 200a is used which results in a distorted test image 201. The 
distorted nature of 2D test image 201 is evident from the relative positions of 
various portions of bottle 201 within the test image, including lower rim 212 
of bottle 202, bottom edge 216 of label 203 and top edge 214 of label 203. The 
entire area of interest lies between bottom edge 216 and top edge 214. 

Brief Summary Text - BSTX (26): 

As shown in FIG. 7C, test image 201 includes a distorted depiction of defect 
204. Due to the significant distortion of the shape and size of defect 204 in 
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the 2D test image 201 shown in FIG. 7C, it is difficult to define defect 
criteria which can be used to accurately identify the existence, size and 
position of defects . The above-described geometric and morphological defect 
criteria are not sufficient for this purpose. 

Brief Summary Text - BSTX (28): 

In performing training, a data structure is created containing information 
important for the subsequent inspection of a test image. The process includes 
acquiring several sample images taken under varying conditions, and then 
calculating statistics from data gathered in connection with the sample images. 
The calculated statistics include a template image which is the arithmetic mean 
of the several sample images. The template image is the image which is 
subtracted from the test image, and may be further used for purposes of 
normalization mapping. The calculated statistics also include a standard 
deviation image. This image may be used to derive a threshold image, which may 
be a linear mapping of the standard deviation image. Other statistics may be 
calculated from the acquired sample images for purposes of calculating a 
normalization map. Such statistics may include a histogram, mean, standard 
deviation, and left and right tails. 

Brief Summary Text - BSTX (32): 

These conditions cannot be met when inspecting for defects (or other 
non-conforming objects) within a semi-opaque enclosure such as a labeled 
bottle. In the conventional labeled bottle inspection system illustrated in 
FIGS. 7A-7C, the defect 204 is on an inner side-wall surface 208 of a bottle 
202, hidden behind a label 203. Sample images, which would necessarily include 
an image of label 203, would not be repeatable. This is because the shape of 
label 203, which forms part of 2D test image 201, changes from bottle to 
bottle, due to typical variations caused when label 203 is applied to the outer 
side- wall surface 210 of bottle 203. In addition, 2D test image 201 is clearly 
distorted due to the use of a wide-angle lens 200a in acquiring the test image. 

Brief Summary Text - BSTX (42): 

A process of distinguishing between different types of regions within an 
image. Object^background discrimination is a particular type of segmentation 
which includes identifying those portions within an image corresponding to an 
object and those portions within the image corresponding to the background. 
Defect and flaw detection is a type of objec^ackground discrimination. 

Brief Summary Text - BSTX (44): 

A process (commonly used in Golden Template Comparison flaw and defect 
detection methods) of obtaining a plurality (usually several) sample images of 
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flawless items to generate both a template image and statistical information 
pertaining to the template image. 

Brief Summary Text - BSTX (48): 

Another object of the present invention is to provide an improved 
segmentation method which can detect a defect (or another object of interest) 
of unknown shape, configuration, and location from within a semi-opaque 
enclosure of varying shape and configuration, such as a labeled bottle. 

Brief Summary Text - BSTX (49): 

A further object of the present invention is to provide an improved machine 
vision-based -inspection -svstem or method which facilitates an accurate 
interpretation of inspection results. More specifically, the machine 
vision-based inspection system or method detects defects with high sensitivity 
but with a decreased occurrence of false alarms, i.e., false "failure" 
indications. 

Brief Summary Text - BSTX (50): 

Other features may be provided to achieve yet additional objects and 
advantages. For example, a simplified template comparison segmentation 
technique may be provided which detects flaws and defects in a semi-opaque 
enclosure such as a labeled bottle. The simplified template comparison 
technique does not require high repeatability of the items being inspected : 
that is, the technique does not require that each item be identical. The 
simplified technique is thus appropriate for accurately 

Brief Summary Text - BSTX (5 1 ): 

detecting defects present on inner side-wall surfaces of labeled bottles or 
other defects within other types of semi-opaque enclosures. 

Brief Summary Text - BSTX (52): 

The simplified template comparison technique does not require training as do 
Golden Template Comparison and similar techniques. A template image is 
obtained almost simultaneously with the obtaining of the test image. More 
specifically, the template image is "dynamic" and corresponds precisely to the 
particular item being inspected at that moment. 

Brief Summary Text - BSTX (53): 

The present invention, therefore, is directed to a method or system for 
detecting the presence of a defect or non-conforming object (at least one) of 
unknown shape, configuration, and location within a semi-opaque enclosure. The 
semi-opaque enclosure may comprise a labeled plastic or glass bottle. 
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Brief Summary Text - BSTX (55): 

A first digital image is captured from first channel light reflected off the 
first external surface of the semi-opaque enclosure. More specifically, the 
first digital image may be a reflected-light image formed with front lighting 
impinging on a front side of the rigid semi-opaque enclosure. A second digital 
image is captured from second channel light navigating the object inside the 
enclosure and emanating from the first external surface of the enclosure. More 
specifically, the second digital image may be a transmitted-light image formed 
with back lighting, which is directed onto a back side of the semi-opaque 
enclosure and thus forms a silhouette image of the object inside the enclosure. 
The silhouette image emanates from the first external surface (e.g., the front 
side ) of the enclosure. A difference image is formed substantially devoid of 
information representing the visible pattern on the first external surface, by 
subtracting one of the first and second digital images from the other. 

Drawing Description Text - DRTX (10): 
FIG. 7A illustrates a conventional labeled-bottle visual inspection system; 

Drawing Description Text - DRTX (12): 

FIG. 7C depicts a 2D test image formed using the conventional labeled-bottle 
visual inspection system of FIG. 7 A; 

Drawing Description Text - DRTX (13): 

FIG. 8 is a perspective view of an illustrative embodiment of a 
labeled-bottle visual inspection system which may be provided in connection 
with the present invention; 

Drawing Description Text - DRTX (14): 

FIG. 9 is a flow chart illustrating a visual inspection process which may be 
performed in connection with the visual inspection system shown in FIG. 8; and 

Detailed Description Text - DETX (6): 

Item 506 comprises part of a complete semi-opaque enclosure, such as an 
envelope or a labeled bottle, or another type of semi-opaque enclosure which 
may contain an object or a defect that is to be detected. In FIGS. 1 and 2, 
item 506 is a portion of an outer side wall of a labeled bottle; remaining 
portions of the labeled bottle have been omitted to simplify the explanation of 
the features depicted. The illustrated label includes the letters "ABC 11 
printed on its outer front surface 508. 

Detailed Description Text - DETX (9): 
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FIG. 3 A illustrates reflected-light image 512, and FIG. 3B illustrates 
transmitted-light image 514. As can be seen in FIGS. 3 A and 3B, each of these 
images includes the same background information, representing background print 
515, while only transmitted-light image 5 14 includes an image of object 5 16. A 
difference image is shown in FIG. 3C. When one of the images shown in FIGS. 3A 
and 3B is subtracted from the other, the background information is removed, and 
only object 516 remains. Thresholding is performed on the gray-level 
difference image 528 shown in FIG. 3C, which causes a binarized image to be 
produced. Thresholding techniques that may be used will be further described 
below. A segmented binary image is thus produced that includes an accurate 
depiction of detected object 516. The binary image may contain a matrix of 
pixels, with defect pixels (e.g., " 1 f s") representing detected object 5 1 6, and 
background (healthy) pixels (e.g., "O's") representing background information. 

Detailed Description Text - DETX (12): 

If a single threshold value is used to binarize difference image 528, errors 
will likely occur in the resulting segmented binary image 520. That is, the 
resulting binary image 520 would not likely represent an accurate segmentation. 
For example, if a single threshold value is set to a high value, for example, a 
value greater than 25, then the resulting segmented binary image 520 would 
include a number of broken/disjointed regions, i.e., regions within object 516 
which would be inaccurately identified as part of the background. This would 
hinder the ability to identify object 516, or otherwise recognize it as a 
defect . If a single threshold value is set to a low value, then the system is 
susceptible to picking up shadows and other noise which are part of the 
background image and not part of object 516. Such shadows and noise can cause 
the apparent shape of object 516, as depicted in the resulting segmented binary 
image 520, to become distorted. 

Detailed Description Text - DETX (23): 

r=s.sup.2 t/1000, where t is a threshold coefficient which may be determined 
empirically. 

Detailed Description Text - DETX (24): 

Referring back to FIG. 4, once local threshold image 526 is obtained, 
thresholding mechanism 530 subtracts it from difference image 528. 
Thresholding mechanism 530 then binarizes the result using a map, such that 
pixels with positive difference values are replaced with a defect pixel (e.g., 
a " 1 ") and pixels with a negative difference value are replaced with a 
background (healthy) pixel (e.g., a "0"). An exemplary resulting segmented 
binary image 520 is illustrated at the bottom of FIG. 4. 
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Detailed Description Text - DETX (26): 
FIG. 5 is a perspective view of a bottle having a brand "A" soda label 
surrounding the bottle's periphery, attached to an outer surface of the 
bottle's side wall. A computer vision system may be utilized to detect whether 
an item located inside a semi-opaque enclosure, such as bottle 540, and (if 
desired) to identify the detected item. In the specific case of bottle 540, a 
soda company may wish to inspect the inside surface of bottle 540 to determine 
whether it contains any unwanted debris or matter. The semi-opaque enclosure 
(a labeled bottle 540 as shown in FIG. 5) allows visible light to pass through 
from one side of the enclosure to the other and to be detected by an image 
acquisition device such as a CCD (Charged Coupled Device). Accordingly, a 
reflected-light image and a transmitted-light image may each be obtained and 
utilized to detect (and optionally also identify) an item located within the 
semi-opaque enclosure. 

Detailed Description Text - DETX (30): 
FIG. 8 is a perspective view of a labeled-bottle visual inspection system 
300. The illustrated system 300 comprises a vision control and processing 
computer system 302, an ensemble of bottles 306x (including the bottles 
306a-306c shown in FIG. 8 and others (not shown)) supported by a conveyor 308, 
and various image acquisition components. A front light 3 12 is positioned in 
alignment with a vertical plane which is parallel to the direction of movement 
of conveyor 308. 

Detailed Description Text - DETX (33): 
As the ensemble of bottles 306x (including the three-bottle set 306a-306c, 
positioned between back light 310 and front light 312) moves along conveyor 
308, transmitted-light and reflected-light images are obtained of the selected 
three-bottle set, each time the bottles are moved along conveyor 308 by a 
distance corresponding to one bottle. The table below illustrates the manner 
in which the necessary images can be obtained in order to inspect the complete 
inside and outside of each bottle. The illustrated labeled bottle inspection 
system 300 is capable of inspecting the complete insides of the bottles, 
including portions which are covered by a semi-opaque label, and is further 
capable of inspecting the outside surfaces of the bottles. In order to do 
this, three sets of images are taken for each bottle. A first set of 
reflected-light and transmitted-light (first and second) digital images may be 
obtained while a bottle is in a first orientation (rotated by O.degree.), a 
second set of such images may be obtained while the same bottle is in a second 
orientation (rotated by 120.degree.), and a third set of such images may be 
obtained while the bottle is in a third orientation (rotated by 240.degree.). 
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Detailed Description Text - DETX (35): 

When a three-bottle set is within the range of the image acquisition system, 
including front light 312, back light 310, and camera 3 1 8, a total 
reflected-light image (first digital image) is obtained which includes an image 
of the entire profile of all three bottles within the three-bottle set. 
Similarly, the transmitted-light image (second digital image) includes the 
complete profile of all three bottles within the three-bottle set. 
Conventional inspection processing methods may be performed to determine 
whether defects exist on the outer surfaces of the bottles. Similarly, in 
order to inspect the inner side- wall surfaces of the bottles at locations 
corresponding to transparent portions of the bottles (i.e., portions of the 
bottles not corresponding to the semi-opaque labels), conventional inspection 
processing methods may be performed. 

Detailed Description Text - DETX (36): 

In order to inspect for defects and non-conforming objects located behind a 
semi-opaque label, the pertinent portions of the overall bottle image (i.e., 
the pertinent portions of the first (reflected-light) digital image and second 
(transmitted-light) digital image) are obtained before subsequent processing is 
performed. 

Detailed Description Text - DETX (37): 

Sensor 3 14 is utilized to generate a signal when the position of a bottle 
corresponds to sensor 3 14 as it is moved by conveyor 308. Vision control and 
processing computer system 302 determines when a certain amount of time passes 
from the point at which the bottle passes sensor 3 14, in order to determine 
when a three-bottle set 306a-306c is in correct alignment with the image 
acquisition system. Accordingly, vision control and processing computer-system 
302 facilitates registration of three-bottle sets with the image acquisition 
system. 

Detailed Description Text - DETX (39): 

The system illustrated in FIG. 8 may be modified and still facilitate 
thorough inspection of the inner side- wall surfaces of labeled bottles. For 
example, one or more cameras may be provided to capture first and second 
images, the bottles may be moved in a different manner, and may be rotated 
through less orientations. For example, two sets of images may be obtained for 
each bottle, by rotating a bottle from a first orientation (at O.degree.) to a 
second orientation (rotated by 90.degree.). In addition, different lighting 
and/or mirror arrangements may be provided to effect the front and back 
lighting needed to produce the transmitted-light and reflected-light images. 
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Detailed Description Text - DETX (40): 

Strobes 304a and 304b are provided to freeze the motion in the obtained 
images. They are controlled to correspond in time with the taking of the 
images by camera 318. This is done in order to prevent blurring in the 
resulting captured images. Movement of a bottle by an amount equivalent to 
one-half or more pixels can significantly blur the captured image. Bottles are 
typically moved in industrial settings at a fast pace, usually at 600 bottles 
per minute. Therefore, it is very important to take steps to prevent blurring 
of the captured images used for inspection . An additional, or alternate, 
measure that can be taken to reduce or eliminate blur is use of a shutter 
camera, which will localize (in time) the visual information which is used to 
capture each image. 

Detailed Description Text - DETX (44): 

FIG. 9 is a flow chart of the image processing which may be performed in 
connection with the visual inspection system illustrated in FIG. 8. A first 
digital image 330 and a second digital image 332 are each obtained for each 
orientation of each bottle. A registration mechanism 334 is provided to 
digitally register first digital image 330 with second digital image 332. 
Registration mechanism may perform such digital registration in accordance with 
a method utilizing two-dimensional cross-correlation of images as is disclosed 
by Jain in chapter 2 of a book entitled "Fundamentals of Digital Image 
Processing," Prentise Hall (1989). Chapter two of that book is hereby 
incorporated herein by reference in its entirety. Other additional or 
alternative digital registration techniques may be used at this point of image 
processing as well. 

Detailed Description Text - DETX (45): 

The registered images are then forwarded to first and second windows 336 and 
338. First and second windows 336 and 338 obtain appropriate portions of the 
overall first and second digital images 330 and 332, for subsequent processing. 
When inspecting a labeled bottle, as illustrated in FIG. 8, the portion of the 
image obtained by the first and second windows 336 and 338 corresponds to the 
portion of bottle covered by a label. After windowing the first and second 
digital images with first and second windows 336 and 338, the resulting 
gray-level windowed first and second digital images may be normalized by first 
and second normalization mechanisms 340 and 342. 

Detailed Description Text - DETX (52): 

Upon receipt of a binarized error image output by binarization mechanism 
348, an interpretation mechanism 350 analyzes the various defect pixels in 
relation to background (healthy) pixels in order to determine whether the 
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inspected item should be categorized as either an acceptance or a failure item. 
The interpretation processing may be implemented by simply counting the number 
of defect pixels present in the error image and/or by utilizing Blob analysis. 
These techniques are well-known in the art, and therefore are not discussed in 
detail herein. 

Detailed Description Text - DETX (55): 
By way of example, the vision control and processing computer system 302 may 
be implemented with a VME bus-based vision system, such as a Cognex 5400. 

Other Reference Publication - OREF (1): 
Automated Vision System for Inspection of IC Pads and Bonds, Sreenivasan et 
al., IEEE Transactions on Components, Hybrids, and Manufacturing Technology, 
pp. 335-338, vol. 16, No. 3, May 1993. 
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